Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BlockifyVR (Virtual Reality) #1732

Open
wants to merge 31 commits into
base: master
Choose a base branch
from

Conversation

Brackets-Coder
Copy link

@Brackets-Coder Brackets-Coder commented Oct 23, 2024

BlockifyVR

This is a Virtual Reality extension I've been working on since January 27th of 2023.

The end goal is to have:

  • Little to no performance overhead with the extension
  • Cross-platform compatibility for all major 6DOF headsets A-frame supports
  • An easy-to-use system that's coherent and efficient
  • Works well with any 3D engine
  • Bug-free enough for optimal user experience without frequent disruptions

Here are some things to note:

  • It includes a local copy of the A-frame JS library, version 1.6.0, MIT License. This is for VR Management, rendering, and ease of development as I'd have to write the entire thing from scratch using WebXR, WebGL, and the Gamepad API otherwise. Setting up the VR boilerplate and stereoscopy and other things would be a pain without it.
  • Gallery thumbnail, documentation, and sample projects are not currently available and will be worked on as I get nearer towards release
  • The rendering uses a plane fixed to the camera which directly mirrors the scratch stage with an unshaded texture each frame. This is relatively performant and works well, but there could be better solutions to look into. The plane is placed 1 meter away from the camera and A-frame uses stereoscopic rendering to reduce eye strain. Note that the plane automatically scales to the size of the camera (display) while attempting to maintain the ratio of the stage. I use this instead of drawing directly to the canvas because it won't let me and I would have to re-write the renderer that's already tailored for VR'
  • The above note means that ONLY THE STAGE LAYERS will be rendered. Any HTML Elements above the stage canvas (reporter monitors, "ask" block, etc.) WILL NOT work in a VR session.
  • Sorry the code's a bit messy, reviewing this extension is probably going to be painful... I should fix that

Upcoming release plan:

v1.0-pre-alpha: Earliest version. Minimal functionality, proof-of-concept version. Not available here. Not open source.

v1.0-alpha: work-in-progress. Expect significant bugs and poor performance. Should not be used.

v1.0-beta: Cross-platform compatibility, major bug fixes, and optimizations have been added.

v1.0-release-candidate: Current version. Feature complete. Meets standards of full release with minimal issues. During this release period, preparations for full release include gallery thumbnail, sample project, documentation, etc.

v1.0: First release. Feature complete and meets all standards of great performance, very few issues, cross-platform compatibility, and intuitive design.

v1.5: Bugfixes and optimizations. Possibly a small feature or change, such as controller vibrations.

v2.0: Big update. Most likely something like hand tracking support for some platforms. Also will include some minor bugfixes and optimizations.

v2.2: Bugfixes and optimizations.

Active Development

v1.0-release-candidate

  • Bug fixes and optimizations
  • Prep for release
  • Sample project
  • Documentation

@Brackets-Coder Brackets-Coder marked this pull request as ready for review October 24, 2024 19:02
@Brackets-Coder Brackets-Coder marked this pull request as draft October 24, 2024 19:02
@Brackets-Coder Brackets-Coder changed the title BlockifyVR (Virtual Reality) Extension New Extension: BlockifyVR (Virtual Reality) Oct 24, 2024
@Brackets-Coder Brackets-Coder changed the title New Extension: BlockifyVR (Virtual Reality) BlockifyVR (Virtual Reality) Oct 24, 2024
@SharkPool-SP SharkPool-SP added the pr: new extension Pull requests that add a new extension label Oct 24, 2024
@hammouda101010
Copy link

that looks cool

@hammouda101010
Copy link

that looks cool

It looks cool on the flat screen right now, but it's not so great when you try it. You'd probably get motion sickness. I'm still working on it.

sadly i dont think that's possible to stop this.

@Xeltalliv
Copy link
Contributor

Does this show the same scratch 3D perspective to both eyes?

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Oct 29, 2024

Does this show the same scratch 3D perspective to both eyes?

A-frame has stereoscopic rendering built in and I've enabled it in the rendering settings, so no. This fixes some issues with visualizing depth, so the eye offset allows your brain to process the image with more clarity. Even if the matrices aren't used in the renderer, it should properly use the correct eye offset. Why do you ask, just out of curiosity? If you'd like I could add a block that disables or enables this feature, in case the Scratcher wants to build that framework themselves, which I'd doubt.

Keep in mind this is still a work in progress and most of the code is many months old so it's not the most efficient way to do VR.

@Xeltalliv
Copy link
Contributor

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Oct 29, 2024

Why do you ask, just out of curiosity?

Yes. At one point I tried to extend my AR extension to support AR headsets and VR as well, but the multiple eye rendering has been the main thing holding me back. I was just curious to know how you solved it.

I believe that there can be two XRViewerPose matrices for each eye if available, according to Mozilla Docs. You might want to look into that again. It'd be great if Augmented Reality could support AR headsets such as the Oculus Quest 3 in addition to just mobile touchscreen devices. There should be a views array that can do this. An example can be seen here. Just a suggestion though. I simply think having AR headsets would be a nice update.

@Brackets-Coder

This comment was marked as resolved.

@Xeltalliv
Copy link
Contributor

Xeltalliv commented Oct 30, 2024

I'll take a look when I have time. Thought it will probably be hard because the only kind of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

@Brackets-Coder

This comment was marked as resolved.

@Brackets-Coder
Copy link
Author

I'll take a look when I have time. Thought it will probably be hard because the only kinf of VR I have access to is phone-based 3DOF no controller VR.

Also, do you mind giving some links, so that I can faster figure out where to look?

As for VR testing, you wouldn't really need a VR headset. I've used Meta Quest's Immersive Web Emulator extension for testing when I didn't care to load up the entire headset.

@Drago-Cuven
Copy link

ah yes, I remember seeing a trailer for this on scratch. I wanted to help, but now I can't

@Drago-Cuven
Copy link

nvm I think I can
i'm Martinelplayz on scratch btw

@siskka7

This comment was marked as resolved.

@Thebloxers998
Copy link

could i try the extension?

Hi siskka7 it's me Thebloxers998

@Thebloxers998
Copy link

Thebloxers998

hmm https://github.com/TurboWarp/extensions/blob/2837a7adfbfc2e3aba49108003686affa1cb7beb/extensions/MasterMath/BlockifyVR.js

I'm not sure what you're "hmm"ing. It's just the source code.

Hi, nice extension

@Xeltalliv
Copy link
Contributor

I'll take a look when I have time.

I took a look and have no idea. It's all code specific to AFRAME. I don't know AFRAME. AFRAME is too big and complicated for me to try to debug, especially if the only tool in my disposal is WebXR API Emulator browser extension.

@Thebloxers998
Copy link

Nic3

@Thebloxers998
Copy link

Nice*

@siskka7
Copy link

siskka7 commented Jan 13, 2025

4. R.js" to download JUST THE EXTENSION. Do NOT download the entire .zip source code of the whole repositor

thank for your very fast response

@Thebloxers998
Copy link

"Ready for review" Yay

@Brackets-Coder Brackets-Coder marked this pull request as draft January 14, 2025 02:35
@siskka7
Copy link

siskka7 commented Jan 14, 2025

it works now .i found on quest 3 that there is down black stripe like it looks like it was flat screen folowing me (no rotation lag) but screen ends down

@siskka7
Copy link

siskka7 commented Jan 14, 2025

i found that when blocks dont work on quest 3 (not big problem for me beceause i can live withow it + we have if blocks)

@siskka7
Copy link

siskka7 commented Jan 15, 2025

i found that when blocks dont work on quest 3 (not big problem for me beceause i can live withow it + we have if blocks)

Could you please specify which blocks weren't working on Quest 3? I'm assuming you're referring to the "When" hat blocks, such as "when key pressed" or "when controller connected".

Please note that most blocks will return nothing at all unless you're in a VR session (e.g., pressing the block to reveal its value in the editor doesn't work). They must be used properly and will only return their proper values after the enter vr mode block has finished executing. The same rules go for the "when" hat blocks.

This is why I needed to provide a sample project, as I haven't made the blocks return any in-editor fallbacks yet.

By my testing, all blocks currently work on quest 2 except the get matrix block.

Oh i inspected only when (key/button) pressed and i was in vr mode not in editor

@siskka7
Copy link

siskka7 commented Jan 15, 2025

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code)
Edit: probably not useful

@siskka7
Copy link

siskka7 commented Jan 15, 2025

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

@siskka7
Copy link

siskka7 commented Jan 15, 2025

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)

probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.

Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

@siskka7
Copy link

siskka7 commented Jan 15, 2025

here is my test project that logs vr input https://github.com/siskka7/siskka7/blob/main/vr%20blockify%20input%20logger.sb3
(tutorial in project) note that this is before was "when block" was fixed i hope this will help developing

@Drago-Cuven
Copy link

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)

https://github.com/immersive-web/body-tracking

@siskka7
Copy link

siskka7 commented Jan 15, 2025

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)
probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.
Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

I believe that the "When button pressed" block issue you're describing is due to a recent change where I made the block inputs allow reporters, but I don't think hat blocks like inputs that allow reporters. I'll get another release out soon that should fix this issue. I'm not sure what you mean by the other blocks though, everything else seems to work fine except the get matrix block

i think there is big misunderstanding i dont thought hat block but (picture1)
Snímek obrazovky 2025-01-15 201559
and the menu looks like (picture2)
Snímek obrazovky 2025-01-15 200909

@siskka7
Copy link

siskka7 commented Jan 15, 2025

3.0 big update idea body tracking (this is just idea. i am pretty sure that this will be hard to code) Edit: probably not useful

Not supported by A-frame (or WebXR as far as I'm aware of)
probably nobody using the extension has a full-body tracking suit and Oculus standalone full-body tracking isn't consistent.

()-rotation of headset
and
()-position of headset
i cant click/switch to any controlers when i open menu there is check icon behind all options (i am on pc and it may be a bug that everything is selected. i did not learned javascript but the check is everywhere but in default blocks there is only one check. the selected one)

No idea what this is about. I haven't really tested on PC VR as my main focus is standalone. The blocks seem to work fine for me and I've never seen multiple menu options selected in a scratch block.
Sometimes pressing menu inputs in the oculus browser with the oculus controllers is a little tricky

i did not mean pc vr but when i was coding this bug showed up (i use github to share projects)

I believe that the "When button pressed" block issue you're describing is due to a recent change where I made the block inputs allow reporters, but I don't think hat blocks like inputs that allow reporters. I'll get another release out soon that should fix this issue. I'm not sure what you mean by the other blocks though, everything else seems to work fine except the get matrix block

i think there is big misunderstanding i dont thought hat block but (picture1) Snímek obrazovky 2025-01-15 201559 and the menu looks like (picture2) Snímek obrazovky 2025-01-15 200909

intriguing. Turbowarp menus aren't supposed to look like that, but I can guarantee it's probably not an extension issue.

so there is no fix idea :(

@siskka7
Copy link

siskka7 commented Jan 16, 2025

Don't know why the "when button" blocks still aren't working... sigh I guess I have another bug to fix cuz it was working earlier
Oh did not mean that

@Thebloxers998
Copy link

Don't know why the "when button" blocks still aren't working... sigh I guess I have another bug to fix cuz it was working earlier

Dang, but how did it end up bugging. Did you check the logic of the block

• Removed redundant pose-matrices component
• Re-wrote all of get matrix block
• Added new camera FOV block
• fixed issues with some blocks & other minor block changes
@Brackets-Coder Brackets-Coder marked this pull request as ready for review January 21, 2025 14:50
@Thebloxers998
Copy link

I guess this is ready for review now?

Yayyyyyyyy!!!!!!!

@Thebloxers998
Copy link

I guess this is ready for review now?

Yayyyyyyyy!!!!!!!

I Happy Now :)

Copy link

@Thebloxers998 Thebloxers998 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have done some nice progress!

@Thebloxers998
Copy link

So....

A-frame doesn't support detecting the menu button/pause button or the Oculus/system button on the oculus-touch-controls component. I tried detecting these manually through the Gamepad API but I guess they're not exposed or something because they don't work and I checked with ChatGPT and it said you couldn't get them.

I'm leaving the "menu" and "system" buttons in the menus of the blocks because they should still work on vive and windows mixed reality

Which speaking of I need people who can test non-Oculus platforms because I haven't really tested them at all.

Other than that I think this extension should be ready for release. I know a lot of them are busy but it'd be great if we could get some reviewers/moderators over here... this extension is moderately large so I expect it to take longer than normal to review

Ok

Copy link

@Thebloxers998 Thebloxers998 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You've done some nice changes 👍

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Jan 29, 2025

I think I'm going to make a few changes to make the code more "professional" and less messy but I can't right now so I'll get around to it when I'm less busy with other things

Likely includes using switch instead of if else in most of the block code and switching to a more object-oriented approach that's easier to work with and reducing the crazy variable situation

@Brackets-Coder
Copy link
Author

Brackets-Coder commented Feb 17, 2025

Still working on cleaning up some of the backend code...

  • Use Scratch.Cast where necessary
  • Use switch case instead of if else where more efficient
  • Use a more object-oriented approach instead of a bajillion variables for easy getting/setting of values and reducing line count
  • Some of this code is two years old and I didn't know what I was doing back then

Although these are easy and probably won't take forever, they're annoying and I'm putting them off for a while in favor of putting more time on a work-in-progress 3D physics extension

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr: new extension Pull requests that add a new extension
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants